Image processing apparatus, image processing method, and image processing system

ABSTRACT

A display image is obtained in accordance with a deviation of a viewpoint position of a driver from a reference viewpoint position, on the basis of a captured image obtained by capturing an image on a rear side from a vehicle. The reference viewpoint position is updated on the basis of a long-term fluctuation of the viewpoint position. For example, the reference viewpoint position is not updated when a line-of-sight of the driver is in a certain region including a display unit that displays a display image. For example, when a viewpoint position of the driver enters a certain region corresponding to a seated state, there is registered, as an initial reference viewpoint position, a viewpoint position of the driver in a case where a line-of-sight of the driver is continuously present for a certain period of time on the display unit that displays a display image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.17/288,092, filed Apr. 23, 2021, which is based on PCT filingPCT/JP2019/048361, filed Dec. 10, 2019, which claims priority to JP2018-232047, filed Dec. 11, 2018, the entire contents of each areincorporated herein by reference.

TECHNICAL FIELD

The present technology relates to an image processing apparatus, animage processing method, and an image processing system, and moreparticularly to an image processing apparatus and the like suitable forapplying to an in-vehicle electronic mirror.

BACKGROUND ART

Conventionally, in-vehicle electronic mirrors have been proposed inwhich rearview mirrors of vehicles (room mirrors and left and right doormirrors) are replaced with cameras and displays. Patent Document 1proposes a technique for varying a range of a camera image to bedisplayed on a display by using a relative position of the driver's headwith respect to a display, in order to solve a difference of appearancein an electronic mirror from appearance in an actual mirror.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2013-216286

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

The technique described in Patent Document 1 can provide a visual fieldthat is intuitive and easily viewable, because a display view angle of arear video image changes in accordance with head motion. However,changes in a posture of a driver may make it impossible to obtain anappropriate rear visual field.

An object of the present technology is to provide a visual field that isintuitive and easily viewable, and to obtain an appropriate rear visualfield even if a posture of a driver changes.

Solutions to Problems

A concept of the present technology is

an image processing apparatus including:

a processing unit configured to obtain a display image in accordancewith a deviation of a viewpoint position of a driver from a referenceviewpoint position, on the basis of a captured image obtained bycapturing an image on a rear side from a vehicle; and

an update unit configured to update the above-described referenceviewpoint position on the basis of a long-term fluctuation of theabove-described viewpoint position.

In the present technology, by the processing unit, a display image isobtained in accordance with a deviation of a viewpoint position of thedriver from a reference viewpoint position, on the basis of a capturedimage obtained by capturing an image on a rear side from the vehicle.Then, the update unit updates the reference viewpoint position on thebasis of a long-term fluctuation of the viewpoint position. For example,the update unit may be made to average, for each fixed interval, theviewpoint position obtained at a predetermined sample rate, andsequentially obtain an update value of the reference viewpoint position.This configuration makes it possible to appropriately obtain the updatevalue of the reference viewpoint position corresponding to the long-termfluctuation of the viewpoint position due to a change in a posture ofthe driver.

Thus, the present technology is for obtaining a display image inaccordance with a deviation of a viewpoint position of the driver from areference viewpoint position, on the basis of a captured image obtainedby capturing an image on a rear side from the vehicle, and for updatingthe reference viewpoint position on the basis of a long-term fluctuationof the viewpoint position. Therefore, it becomes possible to provide avisual field that is intuitive and easily viewable, and to obtain anappropriate rear visual field even if a posture of the driver changes.

Note that, in the present technology, for example, the update unit maybe made not to update the reference viewpoint position when aline-of-sight of the driver is in a certain region including a displayunit that displays a display image. In this case, since a display viewangle of a rear video image changes accurately in response to aviewpoint fluctuation of the driver, it is possible to accuratelyprovide a visual field that is intuitive and easily viewable.

Furthermore, the present technology may further include a registrationunit configured to register, as an initial reference viewpoint position,a viewpoint position of the driver in a case where a line-of-sight ofthe driver is continuously present for a certain period of time on thedisplay unit that displays a display image, for example, when aviewpoint position of the driver enters a certain region correspondingto a seated state. Registering the initial reference viewpoint positionin this way makes it possible to provide an appropriate rear visualfield to the driver from the beginning of driving, even in a case wherethe driver is changed.

Furthermore, in the present technology, for example, the processing unitmay be made to superimpose an image showing the vehicle on the capturedimage, to obtain the display image. In this case, since the displayimage is not only made from the captured image obtained by capturing animage on a rear side from the vehicle, but the display image is obtainedby superimposing the image showing the vehicle on the captured image, itis possible to easily provide a sense of distance by motion parallax.

In this case, for example, the image showing the vehicle may be acomputer graphics image. Using a computer graphics image allows a higherdegree of freedom in generating an image showing the vehicle.

Furthermore, for example, the captured image obtained by capturing animage on a rear side from the vehicle may be a captured image capturedby an image capturing device attached to a rear part of the vehicle, andthe image showing the vehicle may be a vehicle interior image. In thiscase, the display image corresponds to room mirror display. Furthermore,for example, the captured image obtained by capturing an image on a rearside from the vehicle may include a captured image captured by an imagecapturing device attached to a side part of the vehicle, and the imageshowing the vehicle may be a vehicle body image. In this case, thedisplay image corresponds to side mirror display.

Furthermore, for example, the processing unit may be made to change asuperimposed positional relationship between the captured image and theimage showing the vehicle, in accordance with a deviation of a viewpointposition of the driver from a reference viewpoint position. Thisconfiguration can generate motion parallax that is close to that oflooking at an actual rearview mirror, and can assist the driver'sperception of between distances.

In this case, for example, the processing unit may be made to arrangethe captured image and the image showing the vehicle in athree-dimensional space, obtain a virtual viewpoint position thatchanges in accordance with a deviation of a viewpoint position of thedriver from a reference viewpoint position, and convert the capturedimage and the image showing the vehicle into a projected coordinatesystem with a visual field determined by the virtual viewpoint position,to obtain a display image. This configuration makes it possible toaccurately change the superimposed positional relationship between thecaptured image and the image showing the vehicle, in accordance withmotion of a viewpoint of the driver.

Then, in this case, for example, the processing unit may be made toarrange the captured image at a position of a predetermined objectexisting on a rear side from the vehicle. For example, the predeterminedobject may be an object closest to the vehicle, or an object being seenby the driver. By arranging the captured image at a position of thepredetermined object existing on a rear side from the vehicle in thisway, a predetermined object can be arranged with a proper size at aproper position in the three-dimensional space, and the motion parallaxthat occurs between the predetermined object and the image showing thevehicle can be correctly expressed.

Furthermore, for example, the processing unit may be made to superimposethe image showing the vehicle on the captured image to allow thecaptured image to be seen through. This configuration can preventimpairment of rear visibility even when motion parallax is provided bysuperimposing the image showing the vehicle.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view showing an example of component arrangement of avehicle as an embodiment.

FIG. 2 is a view showing a vehicle body (a car body), a vehicle bodyopening (a window), and interior objects of a vehicle.

FIG. 3 is a block diagram showing a configuration example of an imageprocessing apparatus.

FIG. 4 is a view showing component arrangement in a virtual space.

FIG. 5 is a view for explaining viewpoint motion and virtual viewpointmotion.

FIG. 6 is a view showing an example of appearance in an electronicmirror with a reference visual field.

FIG. 7 is a flowchart showing an example of an initialization flow.

FIG. 8 is a view showing an example of a viewpoint detection region anda line-of-sight detection region.

FIG. 9 is a view showing an example of adjustment of a visual fieldsetting by a driver.

FIG. 10 is a view showing an example of viewpoint motion and a movementaverage.

FIG. 11 is a view for explaining a captured camera image.

FIG. 12 is a view showing an arrangement example of a camera image.

FIG. 13 is a view showing a change in appearance of a camera image dueto a difference in arrangement of the camera image.

FIG. 14 is a flowchart showing an example of a processing flow of acamera image arrangement computing unit.

FIG. 15 is a flowchart showing another example of the processing flow ofthe camera image arrangement computing unit.

FIG. 16 is a flowchart showing another example of the processing flow ofthe camera image arrangement computing unit.

FIG. 17 is a view showing camera image arrangement in a virtual space.

FIG. 18 is a view showing an arrangement example of elements necessaryfor drawing in a virtual space.

FIG. 19 is a view showing an example of a display image obtained by animage drawing unit.

FIG. 20 is a flowchart showing an example of processing of the normaloperation flow in the image processing apparatus.

FIG. 21 is a flowchart showing an example of display image processing.

FIG. 22 is a view showing a change in an overlapping degree of drawingobject movement due to viewpoint motion.

FIG. 23 is a view showing variations in drawing processing.

FIG. 24 is a view for explaining conventional electronic mirror(rearview mirror) technology.

FIG. 25 is a view for explaining electronic mirror (rearview mirror)technology of the present technology.

FIG. 26 is a view for explaining a related art regarding a side mirror.

FIG. 27 is a view for explaining an electronic mirror that substitutesfor a side mirror and to which the present technology is applied.

FIG. 28 is a view showing an example of component arrangement of avehicle.

FIG. 29 is a block diagram showing a configuration example of the imageprocessing apparatus.

FIG. 30 is a view showing an arrangement position of a camera image in avirtual space.

FIG. 31 is a view showing an arrangement example of elements necessaryfor drawing in a virtual space.

FIG. 32 is a block diagram showing a configuration example of hardwareof a computer.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an embodiment for implementing the invention (hereinafter,referred to as an embodiment) will be described. Note that thedescription will be given in the following order.

-   -   1. Embodiment    -   2. Modified Example

1. Embodiment

[Component Arrangement of Vehicle]

FIG. 1 shows an example of component arrangement of a vehicle 10 as anembodiment. The vehicle 10 has a vehicle body (a car body) 100, avehicle body opening (a window) 101, and an interior object 102 such asa seat. FIG. 2(a) shows the vehicle body (the car body) 100, a hatchedportion of FIG. 2(b) shows the vehicle body opening (the window) 101,and FIG. 2(c) shows the interior object 102 such as a seat.

Furthermore, the vehicle 10 has a rear image capturing unit 103, a reardistance measuring unit 104, a viewpoint measuring unit 105, and aline-of-sight measuring unit 106. The rear image capturing unit 103 isconfigured by, for example, a complementary metal oxide semiconductor(CMOS) camera, and is attached to a rear-side outer shell of the vehicle10 so as to capture an image on a rear side. The rear distance measuringunit 104 is configured by, for example, a time of flight (ToF) distanceimage sensor, and is attached to the rear-side outer shell of thevehicle 10 so as to acquire a rear distance image.

The viewpoint measuring unit 105 detects a viewpoint position of adriver (a user). The viewpoint measuring unit 105 is attached inside ona front side of the vehicle 10. The viewpoint measuring unit 105includes, for example, a CMOS camera, and measures a position of thedriver's eye as the viewpoint position on the basis of a captured imageof the camera. Note that the viewpoint measuring unit 105 may measurethe viewpoint position of the driver on the basis of, for example, animage captured by an infrared camera. The line-of-sight measuring unit106 detects a line-of-sight of the driver. The line-of-sight measuringunit 106 is attached inside on a front side of the vehicle 10. Theline-of-sight measuring unit 106 includes, for example, a CMOS camera,and detects a line-of-sight of the driver, that is, where the driver islooking, on the basis of an image of the driver's pupil.

Furthermore, the vehicle 10 has a video image display unit (a display)107, a user operation unit 108, and an image processing apparatus 109.The video image display unit 107 is attached inside on a front side ofthe vehicle 10 instead of a conventional room mirror, and has asubstantially rectangular display surface. The video image display unit107 includes a liquid crystal display (LCD), an organic electronicluminescent (EL) panel, and the like.

The user operation unit 108 constitutes a user interface that receivesvarious operations by the driver. This user operation unit 108 includes,for example, a mechanical operation button arranged on an in-frontpanel, and further includes a touch panel arranged on a screen of thevideo image display unit 107, and the like. The video image display unit107 basically displays a rear image of the vehicle 10. However, in acase where a touch panel function is provided, the video image displayunit 107 also displays a user interface (UI) for user operation, ifnecessary.

The image processing apparatus 109 performs processing for obtaining adisplay image to be displayed on the video image display unit 107. Theimage processing apparatus 109 is arranged at any location inside thevehicle 10, for example, in an in-front panel part as illustrated. Theimage processing apparatus 109 obtains a display image in accordancewith a deviation of a viewpoint position of the driver from a referenceviewpoint position, on the basis of a camera image obtained by the rearimage capturing unit 103. Then, in this case, the image processingapparatus 109 updates the reference viewpoint position on the basis of along-term fluctuation of the viewpoint position. This configurationmakes it possible to provide a visual field that is intuitive and easilyviewable, and to obtain an appropriate rear visual field even if aposture of the driver changes.

In this case, the image processing apparatus 109 is not to update thereference viewpoint position when a line-of-sight of the driver is in acertain region including the display unit that displays a display image.This configuration causes a display view angle of a rear video image tobe changed accurately in response to a viewpoint fluctuation of thedriver, which makes it possible to accurately provide a visual fieldthat is intuitive and easily viewable.

Furthermore, in this case, when a viewpoint position of the driverenters a certain region corresponding to a seated state, the imageprocessing apparatus 109 registers, as an initial reference viewpointposition, a viewpoint position of the driver in a case where aline-of-sight of the driver is continuously present for a certain periodof time on the display unit that displays a display image. Thisconfiguration makes it possible to provide an appropriate rear visualfield to the driver from the beginning of driving, even in a case wherethe driver is changed.

Furthermore, in this case, the image processing apparatus 109 obtains adisplay image by superimposing and composing, with 3D CG, a vehicleinterior image (a seat, a headrest, a window, a pillar, and the like) asan image showing the vehicle 10, on a camera image obtained by the rearimage capturing unit 103. In this case, the display image is not madeonly with the camera image, but the display image is obtained bysuperimposing the vehicle interior image on the camera image. Therefore,it is possible to easily provide a sense of distance by motion parallax.

In this case, the image processing apparatus 109 changes a superimposedpositional relationship between the captured image and the vehicleinterior image, in accordance with a deviation of a viewpoint positionof the driver obtained by the viewpoint measuring unit 105 from areference viewpoint position. This configuration allows the driver tohave motion parallax that is close to that of looking at an actual roommirror, and can assist the driver's perception of between distances.

[Configuration of Image Processing Apparatus]

FIG. 3 shows a configuration example of the image processing apparatus109. The image processing apparatus 109 includes a storage unit 111, aview frustum shape position computing unit 112, a body history storageunit 113, a long-term viewpoint motion measuring unit 114, a cameraimage arrangement computing unit 115, a virtual space arrangementcomputing unit 116, a projection computing unit 117, and an imagedrawing unit 118.

As shown in FIG. 4 , in addition to components to be subjected to imageprocessing, that is, 3D CG data of the vehicle 10 (a car body, a window,an interior, and the like), the image processing apparatus 109 arranges,in a virtual space, a camera image obtained by capturing an image on arear side, and places a view frustum obtained on the basis of a virtualviewpoint position and a virtual video image display unit 107A. Then,after performing enlargement/reduction processing, as necessary, theimage processing apparatus 109 outputs an image generated with the viewfrustum as a display image to be displayed on the video image displayunit 107. Note that, in a case where a size of the virtual video imagedisplay unit 107A is the same as that of the video image display unit107, the enlargement/reduction processing is not required.

In this case, as shown in FIG. 5 , the image processing apparatus 109measures, as relative motion with respect to the reference viewpointposition, movement of a viewpoint position of the driver measured by theviewpoint measuring unit 105. In corresponding to this, the imageprocessing apparatus 109 moves the virtual viewpoint position from areference virtual viewpoint position, to change an image (a video image)displayed on the virtual video image display unit 107A and therefore thevideo image display unit 107, and provides the driver with appropriatemotion parallax.

Returning to FIG. 3 , the storage unit 111 stores information regardingthe reference viewpoint position and a reference visual field setting,as well as the 3D CG data of the vehicle. Here, the reference visualfield is a reference rear visual field, and means a visual field withthe view frustum formed by the virtual viewpoint position and thevirtual video image display unit 107A. Therefore, the informationregarding the reference visual field setting is the informationregarding the reference virtual viewpoint position and a position and asize of the virtual video image display unit 107A.

A preferred rear visual field as the reference visual field variesdepending on a driving situation and individuals, but a visual field inwhich top, bottom, left, and right are reflected in a well-balancedmanner and a vanishing point is slightly above a center of the screen isconsidered as a general reference visual field. FIG. 6 shows an exampleof a rear visual field that is preferable as the reference visual field.In this example, the vanishing point is slightly above a center of thescreen in a state of traveling on a straight horizontal road. Note that,in FIG. 6 , an intersection point of broken lines extending in avertical direction and a horizontal direction represents the center ofthe screen.

The image processing apparatus 109 executes an initialization flow afterthe driver (the user) is seated in the driver's seat, and automaticallyregisters an appropriate reference viewpoint position for the driver inthe storage unit 111. The reference viewpoint position registered inthis initialization flow is the initial reference viewpoint position. Byregistering the initial reference viewpoint position in this way, it ispossible to absorb individual differences in a viewpoint position thatshould be a reference for the driver. Further, it becomes possible toprovide an appropriate rear visual field to the driver from thebeginning of driving, even in a case where the driver is changed. Inthis embodiment, the reference viewpoint position is sequentiallyupdated from the initial reference viewpoint position on the basis of along-term fluctuation of the viewpoint position as described later.

A flowchart of FIG. 7 shows an example of the initialization flow. Instep ST11, the image processing apparatus 109 starts processing. Next,in step ST12, the image processing apparatus 109 acquires a currentviewpoint position of the driver on the basis of a detection result ofthe viewpoint measuring unit 105, and also acquires a currentline-of-sight position of the driver on the basis of a detection resultof the line-of-sight measuring unit 106.

Next, in step ST13, the image processing apparatus 109 determineswhether or not the viewpoint is within a viewpoint detection region (seeFIG. 8 ). Here, the viewpoint detection region is set in advance so asto include a viewpoint of the driver in a state of being seated in thedriver's seat. Therefore, the fact that a viewpoint of the driver iswithin this viewpoint detection region can be regarded as the driversitting in the driver's seat.

When the viewpoint is not within the viewpoint detection region, theimage processing apparatus 109 returns to the processing of step ST12.Whereas, when the viewpoint is within the viewpoint detection region,the image processing apparatus 109 determines in step ST14 whether ornot the line-of-sight is on the video image display unit 107 in aline-of-sight detection region (see FIG. 8 ). Here, the line-of-sightdetection region is set in advance in a region including the video imagedisplay unit 107 and a peripheral thereof.

When the line-of-sight is not on the video image display unit 107, theimage processing apparatus 109 returns to the processing of step ST12.Whereas, when the line-of-sight is on the video image display unit 107,the image processing apparatus 109 shifts to the processing of stepST15. In step ST15, the image processing apparatus 109 determineswhether or not the line-of-sight is continuously present on the videoimage display unit 107 for a certain period of time or longer, here forone second or longer.

When the line-of-sight is not on the video image display unit 107continuously present for one second or longer, the image processingapparatus 109 returns to the processing of step ST12. Whereas, when theline-of-sight is continuously present for one second or longer on thevideo image display unit 107, the image processing apparatus 109registers the current viewpoint position as the reference viewpointposition in the storage unit 111 in step ST16. Thereafter, in step ST17,the image processing apparatus 109 ends a series of processing.

The registration of the reference visual field setting can be executedby the driver performing a touch operation on the user operation unit108, for example, the touch panel arranged on the screen, of the videoimage display unit 107. In this case, a visual field setting (a virtualviewpoint position, and the like) is adjusted to obtain desiredappearance of the rear visual field at the reference viewpoint position,and the adjusted visual field setting is registered in the storage unit111 as the reference visual field setting.

FIGS. 9(a) and 9(b) show an example of adjustment of the visual fieldsetting by the driver. This example shows that, when the driver performsa slide operation in a right direction on the touch panel as shown inFIG. 9(a), movement is made from a virtual viewpoint position A to thevirtual viewpoint position B as shown in FIG. 9(b), and a referencevisual field A is changed to a reference visual field B.

Note that the driver seated in the driver's seat does not necessarilyhave to register the reference visual field setting. In a case where thereference visual field setting is not registered, the reference visualfield setting already registered in the storage unit 111 is used as itis. Furthermore, it is also conceivable that the storage unit 111 holdsa plurality of reference visual field settings and manually orautomatically switches the plurality of reference visual field settingsto use depending on a driving situation.

The information regarding the reference viewpoint position and thereference visual field setting stored (registered) in the storage unit111 is used in a normal operation flow, but the reference viewpointposition will be updated sequentially on the basis of a long-termfluctuation of the viewpoint position. Therefore, it is possible toalways obtain an appropriate rear visual field even if a posture of thedriver changes.

Returning to FIG. 3 , in the normal operation flow, the long-termviewpoint motion measuring unit 114 updates the reference viewpointposition stored in the storage unit 111 on the basis of a long-termfluctuation of the viewpoint position of the driver obtained by theviewpoint measuring unit 105. In this case, the long-term viewpointmotion measuring unit 114 averages, for each fixed movement averageinterval, the viewpoint position obtained at a predetermined samplerate, and sequentially obtains an update value of the referenceviewpoint position.

FIG. 10 shows an example of viewpoint motion and a movement average,that is, the update value. The illustrated example shows a case where asample rate of the viewpoint position is 60 sample/sec and a movementaverage interval is 500 msec. Furthermore, the illustrated example isfor only one axis, but it is possible to follow a long-term fluctuationin the viewpoint position, such as a posture fluctuation, and alwaysprovide a favorable rear visual field by performing similar calculationin each of up-down, front-back, and left-right directions and adopting amovement average value as the reference viewpoint position. Note thatthe sample rate of the viewpoint position and the movement averageinterval are not limited to these values.

In this embodiment, the long-term viewpoint motion measuring unit 114does not update the reference viewpoint position when a line-of-sight ofthe driver is within the line-of-sight detection region (see FIG. 8 )including the video image display unit 107. This configuration causes adisplay view angle of a rear video image to be changed accurately inresponse to a viewpoint fluctuation of the driver, which makes itpossible to accurately provide a visual field that is intuitive andeasily viewable. Note that, it is also conceivable to update thereference viewpoint position regardless of whether or not line-of-sightof the driver is within the line-of-sight detection region including thevideo image display unit 107.

Returning to FIG. 3 , the view frustum shape position computing unit 112calculates a shape and a position of the view frustum in the virtualspace, on the basis of information regarding the reference viewpointposition and the reference visual field setting read from the storageunit 111, and the current viewpoint position detected by the viewpointmeasuring unit 105. In this case, a virtual viewpoint position (acurrent virtual viewpoint position) deviated from the reference virtualviewpoint position is obtained (see FIG. 5 ) in accordance with adeviation (a deviation in a distance or a direction) of the viewpointposition (the current viewpoint position) from the reference viewpointposition. Further, on the basis of this virtual viewpoint position andthe size and the position of the virtual video image display unit 107A,a position and a shape of the view frustum with the virtual viewpoint asan apex are obtained (see FIG. 4 ).

The camera image arrangement computing unit 115 calculates anarrangement distance of a camera image in the virtual space, on thebasis of a rear distance image acquired by the rear distance measuringunit 104, a rear camera image acquired by the rear image capturing unit103, the shape and the position of the view frustum obtained by the viewfrustum shape arrangement computing unit 112, and the like. Depending onthis arrangement position of the camera image, appearance (motionparallax) of a subject that is shown in the camera image and appears anddisappears in a vehicle interior image (the car body, the window, theinterior) differs when the driver moves the viewpoint position. In orderto provide appropriate motion parallax, it is necessary to place thecamera image at an appropriate position in the virtual space.

As shown in FIG. 11 , an image actually captured by the camera isobtained by compressing a three-dimensional space in a distancedirection, and bodies (objects) A to D at different distances arecaptured as a two-dimensional image in a size corresponding to thedistance. Therefore, it is not perfectly appropriate by placing thiscamera image anywhere in a three-dimensional space, and a properposition can be obtained only for bodies that are at a distance wherethe camera image is placed. Note that, actually, an image outside adepth of field of a camera lens is blurred, but here, it is consideredas an ideal pan-focus camera.

FIG. 12 shows a case where the camera image is placed at a distance ofthe body A (image arrangement A) and a case where the camera image isplaced at a distance of the body D (image arrangement D). Then, FIGS.13(a), 13(b), and 13(c) show a sight in a case where a visual field(corresponding to a view frustum determined by the virtual viewpointposition) is moved to the right, the center, and the left, respectively.

Comparing FIGS. 13(a), 13(b), and 13(c), a range of the camera imagethat enters the visual field differs between the case of imagearrangement A and the case of image arrangement D. Further, it can beseen that a range of movement in the camera image differs in accordancewith motion of the visual field. This is the motion parallax for a bodyin the camera image. By placing the camera image at a distance of a body(an object) of interest, the motion parallax that occurs between thebody and the vehicle can be correctly expressed.

It should be noted that, for bodies other than the body of interest, adisplayed size and motion parallax caused by the viewpoint motion arenot correctly expressed. In order to provide proper motion parallax forall bodies, it is necessary to capture an image on a rear side in 3D,and separate all bodies to place in the virtual space. However, suchprocessing requires a great deal of calculation power.

The present technology has a feature of providing motion parallax forthe body of interest with a relatively small amount of calculation, bygiving up the motion parallax other than the body of interest.

In order to present a useful sense of distance by the limited motionparallax, it is necessary to select a body of interest suitable forpresenting the sense of distance to the driver. The followings areevents to consider when selecting the body suitable for presenting asense of distance.

-   -   (1) A distance between the vehicle and a body (a body closest to        the vehicle).    -   (2) A change in distance between the vehicle and the body        (whether it is approaching or moving away).    -   (3) A size of the body (It is not necessary to pay attention to        bodies whose size is smaller than a certain level Ex. insects).    -   (4) What the body is (a car, a bicycle, a person, a wall, or a        plant).    -   (5) A thing the driver is looking at (where the driver is        looking).

Ideally, comprehensive determination should be made in consideration ofall of these, but it is possible to provide a useful system even withonly some events. A flowchart of FIG. 14 shows an example of aprocessing flow of the camera image arrangement computing unit 115. Thisprocessing example takes into consideration of the above-mentionedevents (1), (2), and (3), and can be realized by using only a distanceimage acquired by the rear distance measuring unit 104.

The camera image arrangement computing unit 115 executes the processingflow shown in the flowchart of FIG. 14 every time the rear distancemeasuring unit 104 acquires a distance image. Note that the reardistance measuring unit 104 acquires the distance image at a frequencyof, for example, 120 fps.

In step ST21, the camera image arrangement computing unit 115 startsprocessing at a timing when the rear distance measuring unit 104acquires the distance image. Next, in step ST22, the camera imagearrangement computing unit 115 extracts bodies (object) from thedistance image, and creates a list of positions, shapes, sizes, anddistances of bodies having a certain size or larger. Then, in step ST23,the camera image arrangement computing unit 115 stores the created listin the body history storage unit 113.

Next, in step ST24, the camera image arrangement computing unit 115browses history data of the body history storage unit 113, searches forthe same body from the characteristics of the shape, deletes a body withno history from the list, and calculates a relative speed with thevehicle for a body with a history to add to the list.

Next, in step ST25, the camera image arrangement computing unit 115excludes a body that deviates from an effective image capturing distanceof the camera, from the created list. This is intended to remove bodiesthat are at a distance that the camera is out of focus. If the cameraimage cannot be captured even if the distance can be measured, the bodyis inappropriate for a camera image arrangement distance and isexcluded.

Next, in step ST26, the camera image arrangement computing unit 115deletes a body moving away at a certain speed or more, from the list.Next, in step ST27, the camera image arrangement computing unit 115deletes a body that deviates from the view frustum and vicinity thereof,from the list. Then, in step ST28, the camera image arrangementcomputing unit 115 determines whether or not data remains in the list.

When data remains in the list, in step ST29, the camera imagearrangement computing unit 115 adopts a distance to a body closest tothe vehicle, as the camera image arrangement distance. After theprocessing in step ST29, the camera image arrangement computing unit 115ends a series of processing in step ST30.

Furthermore, when no data remains in the list in step ST28, apredetermined default distance is adopted as the camera imagearrangement distance in step ST31. Here, the default distance is adistance suitable for arranging a distant view. In presenting a sense ofdistance, it is desirable to be as far as computing power allows.However, in reality, for example, the default distance is determinedwith reference to the computing power of the rear distance measuringunit 104. For example, the default distance may be about 100 m for alight detection and ranging (LiDAR), and about 250 m for ToF sensor.After the processing in step ST31, the camera image arrangementcomputing unit 115 ends the series of processing in step ST30.

A flowchart of FIG. 15 shows another example of the processing flow ofthe camera image arrangement computing unit 115. This processing exampletakes into consideration of the above-mentioned events (1), (3), and(4), and can be realized by using a camera image obtained by the rearimage capturing unit 103, in addition to a distance image acquired bythe rear distance measuring unit 104.

The camera image arrangement computing unit 115 executes the processingflow shown in the flowchart of FIG. 15 every time the rear distancemeasuring unit 104 acquires a distance image. Note that the reardistance measuring unit 104 acquires the distance image at a frequencyof, for example, 120 fps.

In step ST61, the camera image arrangement computing unit 115 startsprocessing at a timing when the rear distance measuring unit 104acquires the distance image. Next, in step ST62, the camera imagearrangement computing unit 115 extracts bodies from the distance image,and creates a list of positions, shapes, sizes, and distances of bodieshaving a certain size or larger.

Next, in step ST63, the camera image arrangement computing unit 115excludes a body that deviates from an effective image capturing distanceof the camera, from the created list. This is intended to remove bodiesthat are at a distance that the camera is out of focus. If the cameraimage cannot be captured even if the distance can be measured, the bodyis inappropriate for a camera image arrangement distance and isexcluded.

Next, in step ST64, the camera image arrangement computing unit 115recognizes a body by image recognition, and deletes a body unsuitablefor image arrangement (for example, a bird, a dead leaf, and the like)from the list. Next, in step ST65, the camera image arrangementcomputing unit 115 deletes a body that deviates from the view frustumand vicinity thereof, from the list. Then, in step ST66, the cameraimage arrangement computing unit 115 determines whether or not dataremains in the list.

When data remains in the list, in step ST67, the camera imagearrangement computing unit 115 adopts a distance to a body closest tothe vehicle, as the camera image arrangement distance. After theprocessing in step ST67, the camera image arrangement computing unit 115ends a series of processing in step ST68.

Furthermore, when no data remains in the list in step ST66, apredetermined default distance (a distance suitable for arranging adistant view) is adopted as the camera image arrangement distance instep ST69. After the processing in step ST69, the camera imagearrangement computing unit 115 ends the series of processing in stepST68.

A flowchart of FIG. 16 shows still another example of the processingflow of the camera image arrangement computing unit 115. This processingexample takes into consideration of the above-mentioned events (1), (3),and (5), and can be realized by using a line-of-sight detection resultof the driver (the user) by the line-of-sight measuring unit 106, inaddition to a distance image acquired by the rear distance measuringunit 104.

The camera image arrangement computing unit 115 executes the processingflow shown in the flowchart of FIG. 16 every time the rear distancemeasuring unit 104 acquires a distance image. Note that the reardistance measuring unit 104 acquires the distance image at a frequencyof, for example, 120 fps.

In step ST71, the camera image arrangement computing unit 115 startsprocessing at a timing when the rear distance measuring unit 104acquires the distance image. Next, in step ST72, the camera imagearrangement computing unit 115 extracts bodies from the distance image,and creates a list of positions, shapes, sizes, and distances of bodieshaving a certain size or larger.

Next, in step ST73, the camera image arrangement computing unit 115excludes a body that deviates from an effective image capturing distanceof the camera, from the created list. Then, in step ST74, the cameraimage arrangement computing unit 115 determines whether or not dataremains in the list.

When data remains in the list, in step ST75, the camera imagearrangement computing unit 115 acquires a line-of-sight of the driver(the user) obtained by the line-of-sight measuring unit 106. Then, instep ST76, the camera image arrangement computing unit 115 adopts adistance of a body at a position closest to the line-of-sight, as thecamera image arrangement distance. After the processing in step ST76,the camera image arrangement computing unit 115 ends a series ofprocessing in step ST77.

Furthermore, when no data remains in the list in step ST74, apredetermined default distance (a distance suitable for arranging adistant view) is adopted as the camera image arrangement distance instep ST78. After the processing in step ST78, the camera imagearrangement computing unit 115 ends the series of processing in stepST77.

FIG. 17 shows an arrangement position of a camera image in a virtualspace. The camera image has been obtained by capturing with the rearimage capturing unit 103, at a predetermined image capturing view angle.This camera image is arranged in the virtual space at a positionseparated from a rear part of the vehicle 10, by a camera imagearrangement distance calculated by the camera image arrangementcomputing unit 115.

Returning to FIG. 3 , the virtual space arrangement computing unit 116arranges elements necessary for drawing in the virtual space. That is,the virtual space arrangement computing unit 116 arranges, in thevirtual space, 3D CG data of the vehicle 10 (a car body, a window, aninterior, and the like) stored in the storage unit 111. The virtualspace arrangement computing unit 116 also arranges the camera image at aposition of the camera image arrangement distance calculated by thecamera image arrangement computing unit 115, and further arranges a viewfrustum on the basis of a shape and a position calculated by the viewfrustum shape arrangement computing unit 112. FIG. 18 shows anarrangement example of elements necessary for drawing in the virtualspace.

The projection computing unit 117 converts an object in the virtualspace into a projection image, with the virtual video image display unit107A as a projection surface. The image drawing unit 118 performsprocessing for drawing details of the camera image and the 3D CG data onthe projection image obtained by the projection computing unit 117. Theimage drawing unit 118 further performs enlargement/reduction processingfor matching a size of an image to a size of the video image displayunit 107, to output a display image to be supplied to the video imagedisplay unit 107. FIG. 19 shows an example of a display image obtainedby the image drawing unit 118.

A flowchart of FIG. 20 shows an example of processing of the normaloperation flow in the image processing apparatus 109. The imageprocessing apparatus 109 synchronizes the processing shown in theflowchart of FIG. 20 with a detection cycle, for example, 1/60 seconds,of a viewpoint and a line-of-sight in the viewpoint measuring unit 105and the line-of-sight measuring unit 106, and executes the processing inthat cycle.

In step ST40, the image processing apparatus 109 starts processing.Next, in step ST41, the image processing apparatus 109 acquires acurrent viewpoint position on the basis of a detection result of theviewpoint measuring unit 105, and also acquires a current line-of-sightposition on the basis of a detection result of the line-of-sightmeasuring unit 106. Then, the image processing apparatus 109 accumulatesviewpoint position data in step ST42.

Next, in step ST43, the image processing apparatus 109 determineswhether or not the line-of-sight is within the line-of-sight detectionregion (see FIG. 8 ). When the line-of-sight is within the line-of-sightdetection region, the image processing apparatus 109 converts, in stepST44, a difference between the reference viewpoint position and thecurrent viewpoint position into a difference of the virtual viewpointfrom the reference virtual viewpoint position, to calculate a virtualviewpoint position (see FIG. 5 ). Next, in step ST45, the imageprocessing apparatus 109 calculates a shape and a position of the viewfrustum from the virtual viewpoint position. After the processing ofstep ST45, the image processing apparatus 109 proceeds to the processingof step ST46.

Whereas, when the line-of-sight is not within the line-of-sightdetection region in step ST43, the image processing apparatus 109acquires, in step ST47, a long-term viewpoint position and therefore anupdate value of the reference viewpoint position, by averaging viewpointpositions for every fixed movement average interval. Then, in step ST48,the image processing apparatus 109 adopts the long-term viewpointposition acquired in step ST47 as the reference viewpoint position, andstores in the storage unit 111.

Note that step ST47 and step ST48 are executed for every fixed movementaverage interval, for example, every 500 msec. However, it isconceivable to perform steps ST47 and ST48 with the viewpoint andviewpoint detection cycles of the viewpoint measuring unit 105 and theline-of-sight measuring unit 106. In that case, in step ST47, viewpointpositions of a past constant movement average interval are averaged toacquire the long-term viewpoint position.

Next, in step ST49, the image processing apparatus 109 calculates ashape and a position of a view frustum from the reference virtualviewpoint position. In this way, when the line-of-sight is not withinthe line-of-sight detection region, a shape and a position of the viewfrustum are always calculated from the reference virtual viewpointposition without being affected by the current viewpoint position.Therefore, a display image of a rear visual field displayed on the videoimage display unit 107 becomes stable regardless of motion of theviewpoint position of the driver (the user). After the processing ofstep ST49, the image processing apparatus 109 proceeds to the processingof step ST46. The image processing apparatus 109 performs display imageprocessing in step ST46, then returns to the processing in step ST41,and repeats similar processing as described above.

A flowchart of FIG. 21 shows an example of the display image processing.In step ST50, the image processing apparatus 109 starts processing.Next, in step ST51, the image processing apparatus 109 acquires a rearcamera image obtained by the rear image capturing unit 103. Next, instep ST52, the image processing apparatus 109 calculates a camera imagearrangement distance.

Next, in step ST53, the image processing apparatus 109 arranges, in thevirtual space, 3D CG data of the vehicle 10 (the car body, the window,the interior, and the like), a camera image, and a view frustum, whichare the elements necessary for drawing (see FIG. 18 ). Next, in stepST54, the image processing apparatus 109 converts components in thevirtual space into a projected coordinate system, to obtain a projectionimage.

Next, in step ST55, the image processing apparatus 109 performsprocessing for drawing details of the camera image and the 3D CG data onthe projection image, to obtain a display image. Next, in step ST56, theimage processing apparatus 109 outputs the display image to the videoimage display unit 107. After the processing in step ST56, the imageprocessing apparatus 109 ends a series of processing in step ST57.

The image processing apparatus 109 executes processing of theabove-mentioned normal operation flow for every detection cycle of aviewpoint and a line-of-sight. Therefore, in a case where theline-of-sight of the driver (the user) is on the video image displayunit 107, in the display image displayed on the video image display unit107, an overlapping degree of a drawing target object is changedappropriately depending on motion of the viewpoint and a distancebetween with the body of interest on a rear side, that is, anappropriate motion parallax can be obtained. Then, the driver can obtainan appropriate sense of distance with respect to the rear camera image.

FIGS. 22(a), 22(b), and 22(c) show an example of the display imagedisplayed on the video image display unit 107. FIG. 22(a) shows a casewhere a viewpoint position of the driver is at a standard viewpointposition, FIG. 22(b) shows a case where the viewpoint position of thedriver is moved to the right from the standard viewpoint position, andFIG. 22(c) shows a case where the viewpoint position of the driver ismoved to the left from the standard viewpoint position. It can be seenthat an overlapping degree between a vehicle interior CG image and anobject (an automobile) in the camera image changes in accordance withthe viewpoint position of the driver.

Note that, in real mirrors, interior objects and the vehicle body createa blind spot where the rear side cannot be seen. However, in the presenttechnology, by transparently drawing at a time of drawing, or by hidinga part, it is also possible to maintain a wide rear visual field whileassisting perception of a sense of distance by motion parallax. Forexample, FIG. 23(a) is a state where the rear seat as the interiorobject is hidden. Furthermore, FIG. 23(b) shows the vehicle body and therear seat as the interior object with a low transmittance, and FIG.23(c) shows the vehicle body and the rear seat as the interior objectwith a high transmittance.

Of course, if the occurrence of blind spots is not a concern, theinterior object or the vehicle body may be drawn with transmittance of0% to generate and display an image like a real mirror.

Furthermore, the interior object is not limited to the sheet or thelike, and distance perception can be further emphasized by drawing apattern on a window glass, for example. FIG. 23(d) shows a state where ahorizontal line is provided as an object on the window glass.

By performing the processing of the above-mentioned normal operationflow, in a case where line-of-sight of the driver (the user) is not onthe video image display unit 107, the image processing apparatus 109updates the reference viewpoint position on the basis of a long-termfluctuation of the viewpoint position of the driver. Therefore, even ifa posture of the driver changes, the rear visual field can beappropriately displayed continuously on the video image display unit107.

FIGS. 24(a) to 24(c) show conventional electronic mirror (rearviewmirror) technology. FIG. 24(a) shows that, in a case where a posture ofthe driver (the user) is at an appropriate position, the rear visualfield is properly displayed on the video image display unit with thereference visual field. FIG. 24(b) shows that the rear visual fielddisplayed on the video image display unit is directed upward in a casewhere there is a sudden fluctuation (a short-term viewpoint fluctuation)in the viewpoint position of the driver due to the driver's entire bodysinking into the seat. FIG. 24(c) shows that the rear visual fielddisplayed on the video image display unit kept directed upward in a casewhere there is a fluctuation of the viewpoint position of the driver (along-term viewpoint fluctuation) due to continuous sinking of thedriver's entire body into the seat.

FIGS. 25(a) to 25(c) show electronic mirror (rearview mirror) technologyof the present technology. FIG. 25(a) shows that, in a case where aposture of the driver (the user) is at an appropriate position, the rearvisual field is properly displayed on the video image display unit 107with the reference visual field. FIG. 25(b) shows that the rear visualfield displayed on the video image display unit 107 is directed upwardin a case where there is a sudden fluctuation (a short-term viewpointfluctuation) in the viewpoint position of the driver due to the driver'sentire body sinking into the seat. FIG. 25(c) shows that the rear visualfield displayed on the video image display unit 107 is back to thereference visual field and is properly displayed, in a case where thereis a fluctuation of the viewpoint position of the driver (a long-termviewpoint fluctuation) due to continuous sinking of the driver's entirebody into the seat.

As described above, in the vehicle 10 shown in FIG. 1 , the imageprocessing apparatus 109 shown in FIG. 3 obtains a display image inaccordance with a deviation of a viewpoint position of the driver from areference viewpoint position, on the basis of a camera image obtained bycapturing an image on a rear side from the vehicle 10, and updates thereference viewpoint position on the basis of a long-term fluctuation ofthe viewpoint position. Therefore, it becomes possible to display avisual field that is intuitive and easily viewable, and to obtain anappropriate rear visual field even if a posture of the driver changes.

Furthermore, in the vehicle 10 shown in FIG. 1 , when a line-of-sight ofthe driver is in a certain region (a line-of-sight detection region)including the video image display unit 108 that displays a displayimage, the image processing apparatus 109 shown in FIG. 3 does notupdate the reference viewpoint position. Therefore, since a display viewangle of a rear video image changes accurately in response to aviewpoint fluctuation of the driver, it is possible to accuratelyprovide a visual field that is intuitive and easily viewable.

Furthermore, in the vehicle 10 shown in FIG. 1 , the image processingapparatus 109 shown in FIG. 3 registers a viewpoint position of thedriver as an initial reference viewpoint position, in a case where aline-of-sight of the driver is continuously present for a certain periodof time on the video image display unit 107 that displays the displayimage, when the viewpoint position of the driver enters a certain regioncorresponding to a seated state. Therefore, it becomes possible toprovide an appropriate rear visual field to the driver from thebeginning of driving, even in a case where the driver is changed.

Furthermore, in the vehicle 10 shown in FIG. 1 , the image processingapparatus 109 shown in FIG. 3 superimposes a vehicle interior image on acamera image obtained by capturing an image on a rear side from thevehicle 10, and obtains a display image to be displayed on the videoimage display unit 107 that is arranged in place of the conventionalroom mirror. The display image is not only made from the camera imageobtained by capturing an image of a rear side of the vehicle, but thedisplay image is obtained by superimposing the vehicle interior image onthe camera image. Therefore, it is possible to easily provide a sense ofdistance by motion parallax.

Furthermore, in the vehicle 10 shown in FIG. 1 , the image processingapparatus 109 shown in FIG. 3 changes a superimposed positionalrelationship between the camera image and the vehicle interior image, inaccordance with a deviation of a viewpoint position of a viewpoint ofthe driver from a reference viewpoint position. Therefore, it ispossible to generate motion parallax that is close to that of looking atan actual rearview mirror, and can assist the driver's perception ofbetween distances.

Furthermore, in the vehicle 10 shown in FIG. 1 , the image processingapparatus 109 shown in FIG. 3 arranges a camera image and an imageshowing the vehicle 10 in a three-dimensional space, obtains a virtualviewpoint position that changes in accordance with motion of a viewpointof the driver, and converts the camera image and the vehicle interiorimage into a projected coordinate system with a visual field determinedby the virtual viewpoint position, to obtain a display image. Therefore,it is possible to accurately change the superimposed positionalrelationship between the camera image and the vehicle interior image, inaccordance with motion of a viewpoint of the driver.

Furthermore, in the vehicle 10 shown in FIG. 1 , the image processingapparatus 109 shown in FIG. 3 arranges a camera image at a position of abody of interest (an object) existing on a rear side from the vehicle10, to obtain a display image. Therefore, the body of interest can bearranged with a proper size at a proper position in thethree-dimensional space, and motion parallax that occurs between thebody and the vehicle interior image can be correctly expressed.

Note that the effects described in this specification are merelyexamples and are not limited, and additional effects may be present.

2. Modified Example

Note that, the above-described embodiment has shown an example in whichthe present technology is applied to an electronic mirror thatsubstitutes for a room mirror of a vehicle. However, the presenttechnology can also be applied to an electronic mirror that substitutesfor a side mirror of a vehicle. Furthermore, by applying the presenttechnology, not only an electronic mirror for a vehicle, but also anelectronic mirror that is supposed to be used by one person can presenta sense of distance close to that of an actual mirror. Similarly, thepresent technology can be applied to an electronic window instead of anelectronic mirror, on an assumption of being used by one person.

A case where the present technology is applied to an electronic mirrorthat substitutes for a side mirror of a vehicle will be described. FIG.26(a) shows an example of a range of a blind spot in a case where aconventional side mirror or an electronic mirror that substitutes forthe conventional side mirror is used. FIG. 26(b) shows an example of animage reflected in a conventional side mirror. In this case, a sense ofdistance of a body on a rear side can be obtained by a difference in asize of the body on a rear side and motion parallax with the shownvehicle body. FIG. 26(c) shows an example of a display image of anelectronic mirror that substitutes for a conventional side mirror. Inthis case, since motion parallax does not occur even if a viewpointmoves, it is difficult to obtain a sense of distance unlike a realmirror.

FIG. 27(a) shows an example of a range of a blind spot in a case ofusing an electronic mirror that substitutes for a side mirror and towhich the present technology is applied. A driver can also visuallyrecognize behind the own vehicle, which is a blind spot with aconventional side mirror or an electronic mirror that substitutes forthe conventional side mirror. FIGS. 27(b) and 27(c) show an example of adisplay image of an electronic mirror that substitutes for the sidemirror and to which the present technology is applied. In this case,since the own vehicle body is superimposed and drawn by 3D CG, it ispossible to provide a sense of distance with motion parallax.Furthermore, as shown in FIG. 27(c), superimposing the own vehicle bodytransparently allows visual recognition behind the own vehicle.

FIG. 28 shows an example of component arrangement of a vehicle 10. Inthis FIG. 28 , parts corresponding to those in FIG. 1 are given with thesame reference numerals, and detailed description thereof will beomitted as appropriate. The vehicle 10 has a vehicle body (a car body)100, a vehicle body opening (a window) 101, and an interior object 102such as a seat. Furthermore, the vehicle 10 has a rear image capturingunit 103, a rear distance measuring unit 104, a viewpoint measuring unit105, and a line-of-sight measuring unit 106.

Furthermore, the vehicle 10 has a right-side rear image capturing unit103R, a right-side rear distance measuring unit 104R, a left-side rearimage capturing unit 103L, and a left-side rear distance measuring unit104L. The right-side rear image capturing unit 103R and the left-siderear image capturing unit 103L are each configured by, for example, aCMOS camera, and attached to, for example, a conventional side mirrorposition of the vehicle 10 so as to capture a rear image. Furthermore,the right-side rear distance measuring unit 104R and the left-side reardistance measuring unit 104L are each configured by, for example, a ToFdistance image sensor, and attached to, for example, a conventional sidemirror position of the vehicle 10 so as to acquire a rear distanceimage.

Furthermore, the vehicle 10 has a right-side rear video image displayunit (a display) 107R, a left-side rear video image display unit (adisplay) 107L, a user operation unit 108, and an image processingapparatus 109S. The right-side rear video image display unit 107R andthe left-side rear video image display unit 107L are each configured byan LCD, an organic EL panel, or the like, attached to right and leftside positions inside on a front side of the vehicle 10, and have asubstantially rectangular display surface.

The user operation unit 108 constitutes a user interface that receivesvarious operations by the driver. The user operation unit 108 includes,for example, a mechanical operation button arranged on an in-frontpanel, and further includes a touch panel arranged on a screen of theright-side rear video image display unit 107R or the left-side rearvideo image display unit 107L, and the like.

The image processing apparatus 109S performs processing for obtaining adisplay image to be displayed on the right-side rear video image displayunit 107R and the left-side rear video image display unit 107L. Theimage processing apparatus 109S is arranged at any location inside thevehicle 10, for example, in an in-front panel part as illustrated. Theimage processing apparatus 109S obtains a display image by superimposingand composing, with 3D CG, the vehicle body (the car body) as an imageshowing the vehicle 10, on a camera image obtained by capturing an imagewith the rear image capturing unit 103, the right-side rear imagecapturing unit 103R, and the left-side rear image capturing unit 103L.

In this way, the display image is not made only with the camera image,but the display image is obtained by superimposing a vehicle body imageon the camera image. Therefore, it is possible to easily provide a senseof distance by motion parallax. Furthermore, by superimposing thevehicle body image transparently, the driver (the user) can visuallyrecognize an invisible body hidden behind the own vehicle.

FIG. 29 shows a configuration example of the image processing apparatus109S. In this FIG. 29 , parts corresponding to those in FIG. 3 are givenwith the same reference numerals, and detailed description thereof willbe omitted as appropriate. The image processing apparatus 109S includesa storage unit 111, a view frustum shape position computing unit 112, abody history storage unit 113, a long-term viewpoint motion measuringunit 114, a camera image arrangement computing unit 115, a virtual spacearrangement computing unit 116, a projection computing unit (right)117R, a projection computing unit (left) 117L, an image drawing unit(right) 118R, and an image drawing unit (left) 118L.

The image processing apparatus 109S arranges, in a virtual space, acamera image obtained by capturing an image on a rear side, in additionto a component to be image-processed, that is, 3D CG data (a car body,and the like) of the vehicle 10. Then, the image processing apparatus109S obtains a view frustum on the basis of a virtual viewpoint positionand a virtual video image display unit that are related to right-siderear display, performs enlargement/reduction processing on an imagegenerated with this view frustum as necessary, and then outputs as aright-side rear display image to be displayed on the right rear videoimage display unit 107R.

Furthermore, similarly, the image processing apparatus 109S obtains aview frustum on the basis of a virtual viewpoint position and a virtualvideo image display unit that are related to left-side rear display,performs enlargement/reduction processing on an image generated withthis view frustum as necessary, and then outputs as a left-side reardisplay image to be displayed on the left rear video image display unit107L.

In this case, the image processing apparatus 109S measures, as relativemotion with respect to a reference viewpoint position, movement of aviewpoint position of the driver measured by the viewpoint measuringunit 105. In corresponding to this, the image processing apparatus 109Smoves the virtual viewpoint position from a reference virtual viewpointposition, to change an image (a video image) displayed on the right-siderear video image display unit 107R and the left-side rear video imagedisplay unit 107L, and provides the driver with appropriate motionparallax.

The storage unit 111 stores information regarding the referenceviewpoint position and a reference visual field setting related to theright-side rear display and the left-side rear display, as well as 3D CGdata of the vehicle. The image processing apparatus 109 executes aninitialization flow after the driver (the user) is seated in thedriver's seat, and automatically registers an appropriate referenceviewpoint position for the driver in the storage unit 111. Thisreference viewpoint position is sequentially updated from an initialreference viewpoint position, on the basis of a long-term fluctuation ofthe viewpoint position.

The image processing apparatus 109 executes an initialization flow afterthe driver (the user) is seated in the driver's seat, and automaticallyregisters an appropriate reference viewpoint position for the driver inthe storage unit 111. The reference viewpoint position registered inthis initialization flow is the initial reference viewpoint position. Byregistering the initial reference viewpoint position in this way, it ispossible to absorb individual differences in a viewpoint position thatshould be a reference for the driver. Further, it becomes possible toprovide an appropriate rear visual field to the driver from thebeginning of driving, even in a case where the driver is changed.

The long-term viewpoint motion measuring unit 114 updates the referenceviewpoint position stored in the storage unit 111 in the normaloperation flow, on the basis of a long-term fluctuation of a viewpointposition of the driver obtained by the viewpoint measuring unit 105. Theview frustum shape position computing unit 112 calculates a shape and aposition of two view frustums for the right-side rear display and theleft-side rear display in a virtual space, on the basis of informationregarding the reference viewpoint position and the reference visualfield setting read from the storage unit 111, and the current viewpointposition detected by the viewpoint measuring unit 105.

The camera image arrangement computing unit 115 calculates anarrangement distance of a camera image in the virtual space on the basisof: a rear distance image acquired by the right-side rear distancemeasuring unit 104R, the rear distance measuring unit 104, and theleft-side rear distance measuring unit 104L; a rear camera imageacquired by the right-side rear image capturing unit 103R, the rearimage capturing unit 103, and the left-side rear image capturing unit103L; the shape and the position of the two view frustums for theright-side rear display and the left-side rear display obtained by theview frustum shape arrangement computing unit 112; and the like.

Depending on this arrangement position of the camera image, appearance(motion parallax) of a subject that is shown in the camera image andappears and disappears in the vehicle body (the car body) differs whenthe driver moves the viewpoint position. In this case, for allowing anappropriate motion parallax to be provided for a body of interest, adistance to the body is calculated as an arrangement distance. Here, acase is also assumed in which the body of interest is different for theright-side rear display and the left-side rear display. In that case,the arrangement distance of the camera image is calculated to bedifferent values for the right-side rear display and the left-side reardisplay.

FIG. 30 shows an arrangement position of a camera image in a virtualspace. The camera image has been obtained by capturing an image with theright-side rear image capturing unit 103R, the rear image capturing unit103, and the left-side rear image capturing unit 103L, at apredetermined image capturing view angle. This camera image is arrangedin the virtual space at a position separated from a rear part of thevehicle 10, by a camera image arrangement distance calculated by thecamera image arrangement computing unit 115.

The virtual space arrangement computing unit 116 arranges elementsnecessary for drawing in the virtual space. That is, the virtual spacearrangement computing unit 116 arranges, in the virtual space, 3D CGdata of the vehicle 10 (the car body, and the like) stored in thestorage unit 111. The virtual space arrangement computing unit 116 alsoarranges the camera image at a position of the camera image arrangementdistance calculated by the camera image arrangement computing unit 115,and further arranges two view frustums for the right-side rear displayand the left-side rear display on the basis of a shape and a positioncalculated by the view frustum shape arrangement computing unit 112.FIG. 31 shows an arrangement example of elements necessary for drawingin the virtual space.

Returning to FIG. 29 , the projection computing unit (right) 117Rconverts an object in the virtual space into a projection image, withthe virtual video image display unit 107A on a right side as aprojection surface. The image drawing unit (right) 118R performsprocessing for drawing details of the camera image and the 3D CG data onthe projection image obtained by the projection computing unit 117R. Theimage drawing unit (right) 118R further performs enlargement/reductionprocessing for matching a size of an image to a size of the right-siderear video image display unit 107R, to output a display image to besupplied to the right-side rear video image display unit 107R.

Furthermore, the projection computing unit (left) 117L converts anobject in the virtual space into a projection image, with the virtualvideo image display unit 107A on a left side as a projection surface.The image drawing unit (left) 118L performs processing for drawingdetails of the camera image and the 3D CG data on the projection imageobtained by the projection computing unit 117L. The image drawing unit(left) 118L further performs enlargement/reduction processing formatching a size of an image to a size of the left-side rear video imagedisplay unit 107L, to output a display image to be supplied to theleft-side rear video image display unit 107L.

Since a basic processing flow in the image processing apparatus 109S issimilar to a processing flow of the image processing apparatus 109 inthe above-described embodiment except that the left and right displayunits are processed separately, the description thereof will be omittedhere.

Note that the series of processing in the image processing apparatuses109 and 109S described above can be executed by hardware or alsoexecuted by software. In a case where the series of processing isperformed by software, a program that configures the software isinstalled in a computer. Here, examples of the computer include, forexample, a computer that is built in dedicated hardware, ageneral-purpose personal computer that can perform various functions bybeing installed with various programs, and the like.

FIG. 32 is a block diagram showing a configuration example of hardwareof a computer 400 that executes the series of processing described abovein accordance with a program.

In the computer 400, a central processing unit (CPU) 401, a read onlymemory (ROM) 402, and a random access memory (RAM) 403 are mutuallyconnected by a bus 404.

The bus 404 is further connected with an input/output interface 405. Tothe input/output interface 405, an input unit 406, an output unit 407, arecording unit 408, a communication unit 409, and a drive 410 areconnected.

The input unit 406 includes an input switch, a button, a microphone, animage sensor, and the like. The output unit 407 includes a display, aspeaker, and the like. The recording unit 408 includes a hard disk, anon-volatile memory, and the like. The communication unit 409 includes anetwork interface or the like. The drive 410 drives a removable medium411 such as a magnetic disk, an optical disk, a magneto-optical disk, ora semiconductor memory.

In the computer 400 configured as described above, the series ofprocessing described above is performed, for example, by the CPU 401loading the program recorded in the recording unit 408 into the RAM 403via the input/output interface 405 and the bus 404, and executing.

The program executed by the computer 400 (the CPU 401) can be providedby being recorded on, for example, the removable medium 411 as a packagemedium or the like. Furthermore, the program can be provided via a wiredor wireless transmission medium such as a local area network, theInternet, or digital satellite broadcasting.

In the computer, by attaching the removable medium 411 to the drive 410,the program can be installed in the recording unit 408 via theinput/output interface 405. Furthermore, the program can be received bythe communication unit 409 via a wired or wireless transmission medium,and installed in the recording unit 408. Besides, the program can beinstalled in advance in the ROM 402 and the recording unit 408.

Note that the program executed by the computer may be a program thatperforms processing in a time series according to an order described inthis specification, or may be a program that performs processing inparallel or at necessary timing such as when a call is made.

Furthermore, although the preferred embodiment of the present disclosurehas been described above in detail with reference to the accompanyingdrawings, the technical scope of the present disclosure is not limitedto such an example. It is obvious that those with ordinary skill in thetechnical field of the present disclosure can arrive various variationsor modifications within the scope of the technical idea described in theclaims, and it is naturally understood that these also fall within thetechnical scope of the present disclosure.

Furthermore, the present technology can also have the followingconfigurations.

(1) An image processing apparatus including:

-   -   a processing unit configured to obtain a display image in        accordance with a deviation of a viewpoint position of a driver        from a reference viewpoint position, on the basis of a captured        image obtained by capturing an image on a rear side from a        vehicle; and    -   an update unit configured to update the above-described        reference viewpoint position on the basis of a long-term        fluctuation of the above-described viewpoint position.

(2) The image processing apparatus according to (1) above, in which

-   -   the above-described update unit does not update the        above-described reference viewpoint position when a        line-of-sight of the above-described driver is in a certain        region including a display unit that displays the        above-described display image.

(3) The image processing apparatus according to (1) or (2) above,further including:

-   -   a registration unit configured to register, as an initial        reference viewpoint position, a viewpoint position of the        above-described driver in a case where a line-of-sight of the        above-described driver is continuously present for a certain        period of time on the display unit that displays the        above-described display image, when a viewpoint position of the        above-described driver enters a certain region corresponding to        a seated state.

(4) The image processing apparatus according to any one of (1) to (3)above, in which

-   -   the above-described update unit averages, for each fixed        interval, the above-described viewpoint position obtained at a        predetermined sample rate, and sequentially obtains an update        value of the above-described reference viewpoint position.

(5) The image processing apparatus according to any one of (1) to (4)above, in which

-   -   the above-described processing unit superimposes an image        showing the above-described vehicle on the above-described        captured image, to obtain the above-described display image.

(6) The image processing apparatus according to (5) above, in which

-   -   an image showing the above-described vehicle is a computer        graphics image.

(7) The image processing apparatus according to (5) or (6) above, inwhich

-   -   a captured image obtained by capturing an image on a rear side        from the above-described vehicle is a captured image captured by        an image capturing device attached to a rear part of the        above-described vehicle, and    -   an image showing the above-described vehicle is a vehicle        interior image.

(8) The image processing apparatus according to (5) or (6) above, inwhich

-   -   a captured image obtained by capturing an image on a rear side        from the above-described vehicle includes a captured image        captured by an image capturing device attached to a side part of        the above-described vehicle, and    -   an image showing the above-described vehicle is a vehicle body        image.

(9) The image processing apparatus according to any one of (5) to (8)above, in which

-   -   the above-described processing unit changes a superimposed        positional relationship between the above-described captured        image and the image showing the above-described vehicle, in        accordance with a deviation of a viewpoint position of the        above-described driver from a reference viewpoint position.

(10) The image processing apparatus according to (9) above, in which

-   -   the above-described processing unit    -   arranges the above-described captured image and the image        showing the above-described vehicle in a three-dimensional        space, and    -   obtains a virtual viewpoint position that changes in accordance        with a deviation of a viewpoint position of the above-described        driver from a reference viewpoint position, and converts the        above-described captured image and the image showing the        above-described vehicle into a projected coordinate system with        a visual field determined by the virtual viewpoint position, to        obtain the above-described display image.

(11) The image processing apparatus according to (10) above, in which

-   -   the above-described processing unit arranges the above-described        captured image at a position of a predetermined object existing        on a rear side from the above-described vehicle.

(12) The image processing apparatus according to (11) above, in which

-   -   the predetermined object is an object closest to the        above-described vehicle.

(13) The image processing apparatus according to (11) above, in which

-   -   the predetermined object is an object being seen by the        above-described driver.

(14) The image processing apparatus according to any one of (5) to (13)above, in which

-   -   the above-described processing unit superimposes the image        showing the above-described vehicle on the above-described        captured image to allow the captured image to be seen through.

(15) An image processing method including:

-   -   a procedure for obtaining a display image in accordance with a        deviation of a viewpoint position of a driver from a reference        viewpoint position, on the basis of a captured image obtained by        capturing an image on a rear side from a vehicle; and    -   a procedure for updating the above-described reference viewpoint        position on the basis of a long-term fluctuation of the        above-described viewpoint position.

(16) An image processing system including:

-   -   an image capturing unit configured to capture an image on a rear        side from a vehicle;

a processing unit configured to obtain a display image in accordancewith a deviation of a viewpoint position of a driver from a referenceviewpoint position, on the basis of a captured image obtained bycapturing with the above-described image capturing unit;

-   -   an update unit configured to update the above-described        reference viewpoint position on the basis of a long-term        fluctuation of the above-described viewpoint position; and    -   a display unit configured to display a display image obtained by        the above-described processing unit.

(17) A program for causing a computer to function as:

-   -   processing means configured to obtain a display image in        accordance with a deviation of a viewpoint position of a driver        from a reference viewpoint position, on the basis of a captured        image obtained by capturing an image on a rear side from a        vehicle; and update means configured to update the        above-described reference viewpoint position on the basis of a        long-term fluctuation of the above-described viewpoint position.

REFERENCE SIGNS LIST

-   10 Vehicle-   100 Vehicle body (car body)-   101 Vehicle body opening (window)-   102 Interior object-   103 Rear image capturing unit-   103R Right-side rear image capturing unit-   103L Left-side rear image capturing unit-   104 Rear distance measuring unit-   104R Right-side rear distance measuring unit-   104L Left-side rear distance measuring unit-   105 Viewpoint measuring unit-   106 Line-of-sight measuring unit-   107 Video image display unit-   107A Virtual video image display unit-   107R Right-side rear video image display unit-   107L Left-side rear video image display unit-   108 User operation unit-   109, 109S Image processing apparatus-   111 Storage unit-   112 View frustum shape arrangement computing unit-   113 Body history storage unit-   114 Long-term viewpoint motion measuring unit-   115 Camera image arrangement computing unit-   116 Virtual space arrangement computing unit-   117 Projection computing unit-   117R Projection computing unit (right)-   117L Projection computing unit (left)-   118 Image drawing unit-   118R Image drawing unit (right)-   118L Image drawing unit (left)

The invention claimed is:
 1. An image processing apparatus comprising:circuitry configured to obtain a display image in accordance with adifference between a viewpoint position of a driver and a referenceviewpoint position, on a basis of a captured image obtained by capturinga rearward image from a vehicle; and update the reference viewpointposition on a basis of a long-term fluctuation of the viewpoint positionof the driver, wherein the updated reference viewpoint position is usedto obtain the display image.
 2. The image processing apparatus accordingto claim 1, wherein the circuitry does not update the referenceviewpoint position when a line-of-sight of the driver is in a certainregion including a display that displays the display image.
 3. The imageprocessing apparatus according to claim 1, wherein the circuitry isconfigured to register, as an initial reference viewpoint position, aviewpoint position of the driver in a case where a line-of-sight of thedriver is continuously present for a certain period of time on thedisplay that displays the display image, when a viewpoint position ofthe driver enters a certain region corresponding to a seated state. 4.The image processing apparatus according to claim 1, wherein thecircuitry is configured to: obtain a plurality of the viewpointpositions of the driver at a predetermined sample rate for apredetermined time interval; calculate an average of the plurality ofthe viewpoint positions; and update the reference viewpoint positionbased on the calculated average.
 5. The image processing apparatusaccording to claim 1, wherein the circuitry is configured to superimposean image showing the vehicle on the captured image, to obtain thedisplay image.
 6. The image processing apparatus according to claim 5,wherein an image showing the vehicle includes a computer graphics image.7. The image processing apparatus according to claim 5, wherein acaptured image obtained by capturing an image on a rear side from thevehicle includes a captured image captured by an image sensor attachedto a rear part of the vehicle, and an image showing the vehicle includesa vehicle interior image.
 8. The image processing apparatus according toclaim 5, wherein a captured image obtained by capturing an image on arear side from the vehicle includes a captured image captured by animage capturing device attached to a side part of the vehicle, and animage showing the vehicle includes a vehicle body image.
 9. The imageprocessing apparatus according to claim 5, wherein the circuitry isconfigured to change a superimposed positional relationship between thecaptured image and the image showing the vehicle, in accordance with adeviation of a viewpoint position of the driver from a referenceviewpoint position.
 10. The image processing apparatus according toclaim 9, wherein the circuitry is configured to: arrange the capturedimage and the image showing the vehicle in a three-dimensional space,and obtain a virtual viewpoint position that changes in accordance witha deviation of a viewpoint position of the driver from a referenceviewpoint position, and convert the captured image and the image showingthe vehicle into a projected coordinate system with a visual fielddetermined by the virtual viewpoint position, to obtain the displayimage.
 11. The image processing apparatus according to claim 10, whereinthe circuitry is configured to arrange the captured image at a positionof a predetermined object existing on a rear side from the vehicle. 12.The image processing apparatus according to claim 11, wherein thepredetermined object includes an object closest to the vehicle.
 13. Theimage processing apparatus according to claim 11, wherein thepredetermined object includes an object being seen by the driver. 14.The image processing apparatus according to claim 5, wherein thecircuitry is configured to superimpose the image showing the vehicle onthe captured image to allow the captured image to be seen through. 15.The image processing apparatus according to claim 1, wherein thecircuitry is configured to update the reference viewpoint position on abasis of a plurality of the viewpoint positions of the driver within apredetermined time period.
 16. An image processing method comprising:obtaining, using circuitry, a display image in accordance with adifference between a viewpoint position of a driver and a referenceviewpoint position, on a basis of a captured image obtained by capturinga rearward image from a vehicle; and updating, using the circuitry, thereference viewpoint position on a basis of a long-term fluctuation ofthe viewpoint position of the driver, wherein the updated referenceviewpoint position is used to obtain the display image.
 17. An imageprocessing system comprising: an image sensor configured to capture arearward image from a vehicle; circuitry configured to obtain a displayimage in accordance with a difference between a viewpoint position of adriver and a reference viewpoint position, on a basis of a capturedimage captured by the image capturing unit, and update the referenceviewpoint position on a basis of a long-term fluctuation of theviewpoint position; and a display configured to display a display imageobtained by the circuitry, wherein the updated reference viewpointposition is used to obtain the display image.